Mixed environment display of attached data

ABSTRACT

A holographic user interface may display status information in proximity to relevant components of the computing device, such as a robot, allowing a user to readily associate the status information with the relevant components. Arrangements of the graphical displays may utilize graphical elements to show an association between any displayed data and any component of a computing device. Based on data indicating the size, shape, and configuration of a robot&#39;s physical parts, techniques disclosed herein can arrange displayed status data, which may involve a holographic UI, in a relevant context. In addition, techniques disclosed herein allow a user to edit data, or provide an input to one or more computing devices in response to the display of any status data.

CROSS REFERENCE TO RELATED APPLICATION

This application is non-provisional application of co-pending U.S.Application No. 62/153,460 filed Apr. 27, 2015 entitled “EnhancedConfiguration and Control of Robots,” which is expressly incorporatedherein by reference.

BACKGROUND

Mobile robots need to be able to operate within an environment to avoidobstacles, perform tasks, identify paths from one location to another,etc. Existing technologies require complex and expensive hardware aswell as complex and computationally expensive software order to performmost tasks and navigate effectively. Among many other practicaldrawbacks, existing systems place most robotic functions out of reach oflow to mid-range robots.

Among other drawbacks of some existing technologies, when working withrobots or other electronic or mechanical devices, it can be difficult orcostly to add display panels and input mechanisms for processing,monitoring and communicating certain types of information. For instance,it may be difficult in some systems to track a status, setting orconfiguration modification to a robot's software.

Some solutions to this drawback include the utilization of displaypanels and input keypads. However, such solutions add complexity andcost to any robot or computing device. In other solutions, a robot mayoperate a web server and enable a view of data via the web browser on aremote computer. Such solutions, while cheaper than physical displaysand input panels, do not provide a granular level of context with anycommunicated information. Many developments in this area have been madebut most existing systems are not cost effective, particularly when itcomes to low to mid-range devices. It also may be difficult to configurea robot with new instructions or operation procedures.

SUMMARY

Technologies described herein provide a mixed environment display ofattached data. In some configurations, techniques disclosed hereininvolve a first computing device having one or more sensors configuredto detect a location and geometric parameters of a second computingdevice and an environment surrounding the second computing device. Insome configurations, for example, the first computing device may be apersonal computer or a head-mounted display (HMD), and the secondcomputing device may be a mobile robot. The first computing device canbe configured to receive status data indicating a status associated withthe second computing device. For example, the status data may include atemperature of a component, usage data, performance data, location data,or any other information related to the second computing device or acomponent of the second computing device. The status data can begenerated by processing signals of one or more sensors, cameras or otherinput devices of the first computing device. In some configurations, thestatus data can be communicated directly from the second computingdevice.

Based on one or more factors, the system determines if the status datais to be displayed. For example, a status associated with the secondcomputing device can be displayed if the obtained status data indicatesthat at least one value, such as temperature of a component, reaches athreshold. In another example, a status associated with the secondcomputing device can be displayed if an error or conflict, e.g., acollision, is detected. Status data associated with any computing devicecan be displayed and based on any number of factors. In yet anotherexample, a status can be displayed if a changed circumstance isdetected. Such an example may involve the display of data based on amovement or changed state of one or more objects near or attached to thesecond computing device.

Once the system determines if the status data is to be displayed, thesystem generates and displays a graphical element describing at least aportion of the status data on an interface of the first computingdevice. In some configurations, the interface is configured to provide areal-world view of at least a portion of the second computing devicethrough a section of the interface. The graphical element describing thestatus data can be configured to indicate an association between thegraphical element describing the status data and at least the portion ofthe real-world view of the second computing device. In someconfigurations, a display may be configured to indicate an associationbetween the graphical element describing the status data and a renderingof the second computing device.

By the use of the techniques disclosed herein, a holographic userinterface may display status information in proximity to relevantcomponents of a computing device, such as a robot, allowing a user toreadily associate the status information with the relevant components.Arrangements of the graphical displays may utilize graphical elements toshow an association between any displayed data and any object. Based ondata indicating the size, shape, and configuration of a robot's physicalparts, techniques disclosed herein can arrange displayed status data ina relevant context. In addition, techniques disclosed herein allow auser to edit data, or provide an input to one or more computing devicesin response to the display of status data or other data.

It should be appreciated that the above-described subject matter mayalso be implemented as a computer-controlled apparatus, a computerprocess, a computing system, or as an article of manufacture such as acomputer-readable medium. These and various other features will beapparent from a reading of the following Detailed Description and areview of the associated drawings.

This Summary is provided to introduce a selection of concepts in asimplified form that are further described below in the DetailedDescription. This Summary is not intended to identify key features oressential features of the claimed subject matter, nor is it intendedthat this Summary be used to limit the scope of the claimed subjectmatter. Furthermore, the claimed subject matter is not limited toimplementations that solve any or all disadvantages noted in any part ofthis disclosure.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram showing several example components forproviding enhanced control of one or more robots, a mixed environmentdisplay of robotic actions, and a mixed environment display of attacheddata;

FIGS. 2A-2B illustrate example scenario of a robot performing a numberof tasks based on a set of instructions;

FIGS. 3A-3E illustrate example renderings of model data showing a robotperforming a number of tasks based on a set of instructions;

FIG. 4A illustrates example rendering of one or more displays of dataattached to one or more objects;

FIG. 4B illustrates example rendering of a graphical element of attacheddata indicating movement of a computing device;

FIG. 4C illustrates example rendering of a graphical element of dataattached to a real-world view of a computing device;

FIG. 4D illustrates example rendering of data attached to a real-worldview of a computing device with the display of a mixed environmentaugmentation of the computing device;

FIG. 5 is flow a diagram illustrating a routine that may be used forproviding mixed environment display of robotic actions;

FIG. 6 is flow a diagram illustrating a routine that may be used forproviding enhanced configuration and control of robots;

FIG. 7 is flow a diagram illustrating a routine that may be used forproviding a mixed environment display of attached data;

FIG. 8 is a computer architecture diagram illustrating an illustrativecomputer hardware and software architecture for a computing systemcapable of implementing aspects of the techniques and technologiespresented herein;

FIG. 9 is a diagram illustrating a distributed computing environmentcapable of implementing aspects of the techniques and technologiespresented herein; and

FIG. 10 is a computer architecture diagram illustrating a computingdevice architecture for a computing device capable of implementingaspects of the techniques and technologies presented herein.

DETAILED DESCRIPTION

Technologies described herein provide enhanced control of one or morerobots, a mixed environment display of robotic actions, and a mixedenvironment display of attached data. Using a computing device, such asa head-mounted display (an “HMD”), equipped with a number ofcapabilities, such as Surface Reconstruction capabilities coupled withtechniques of localizing a robot in an environment, the techniquesprovided herein cause a robot to have environment understanding,obstacle avoidance and pathfinding. Based on the information received bythe computing device, instructions, such as pathfinding instructions,may be generated. The instructions may be communicated to the robot forexecution.

In some configurations, as the robot follows the instructions, thecomputing device can continue to track the robot's location and theaccuracy with which the robot is following the specified path. If therobot starts to diverge from the path, the computing device can detectsuch actions and send corrective instructions to get the robot back ontrack. Similarly, if the environment changes, e.g., a new obstructionappears, invalidating the path sent to the robot, those instructions canbe corrected to allow the robot to react to the environment dynamically.

Technologies described herein provide techniques for visualizing arobot's actions in the space where they will occur before they areperformed for improved understanding and safety. In addition, techniquesand technologies provide visualizations from the perspective of a robot(of the space around the robot), e.g., the location and identificationof people or objects, for improved understanding and safety. Techniquesand technologies also provide visualizations of past actions of a robotin the space where they occurred to confirm correct operation and/ordebug incorrect operation.

Technologies described herein provide techniques for attachingholographic UIs and other graphical elements to display informationassociated with a robot and environmental elements, such as objectsaround the robot. The generation of a holographic or other graphicalelement associates the information attached to the robot or otherobjects. The techniques also provide the ability for a user to editinstructions, information and other data displayed to the user.Techniques disclosed herein attach the holograph or other graphicalelements representing information to the robot or other objects orelectronic devices where it is most relevant. By providing graphicalrepresentation association information with one or more components orobjects, a user may readily understand a status, scenario or a broadercontext associated with the robot and its surroundings.

Technologies described herein provide techniques for capturing andinterpreting user actions such as a gesture, voice command or any othertype of input. User actions can be interpreted to generate datadescribing the actions, such as a gaze direction, and the datadescribing the actions may be interpreted to generate instructions forcausing a robot to perform the actions of the user. The instructions arebased on a number of other factors, such as objects or environmentalelements surrounding the robot. By localizing the robot with a computingdevice configured to collect and process contextual data, a user caneasily and naturally specify locations in space that can be translatedinto locations that the robot understands.

As summarized above, when working with robots, it may be difficult todetermine what a robot is planning and what it may or may not understandabout its surroundings. Technologies and techniques provided hereingenerate a mixed reality display configured to enable a user tonaturally visualize a robot's planned actions in the space where theywill occur as well as its perception of the world around it, enablingsafer and more natural interaction with the robot.

It should be appreciated that the above-described subject matter may beimplemented as a computer-controlled apparatus, a computer process, acomputing system, or as an article of manufacture such as acomputer-readable storage medium. These and various other features willbe apparent from a reading of the following Detailed Description and areview of the associated drawings. Furthermore, the claimed subjectmatter is not limited to implementations that solve any or alldisadvantages noted in any part of this disclosure.

While the subject matter described herein is primarily presented in thegeneral context of techniques for providing enhanced control of one ormore robots, a mixed environment display of robotic actions, and a mixedenvironment display of attached data, it can be appreciated that thetechniques described herein may apply to any type of sensor and/or anytype of device embodying the sensors. As will be described in moredetail herein, it can be appreciated that implementations of thetechniques and technologies described herein may include the use ofsolid state circuits, digital logic circuits, computer component, and/orsoftware executing on one or more devices. Signals described herein mayinclude analog and/or digital signals for communicating a changed state,movement and/or any data associated with motion detection.

While the subject matter described herein is presented in the generalcontext of program modules that execute in conjunction with theexecution of an operating system and application programs on a computersystem, those skilled in the art will recognize that otherimplementations may be performed in combination with other types ofprogram modules. Generally, program modules include routines, programs,components, data structures, and other types of structures that performparticular tasks or implement particular abstract data types. Moreover,those skilled in the art will appreciate that the subject matterdescribed herein may be practiced with other computer systemconfigurations, including hand-held devices, multiprocessor systems,microprocessor-based or programmable consumer electronics,minicomputers, mainframe computers, and the like.

In the following detailed description, references are made to theaccompanying drawings that form a part hereof, and in which are shown byway of illustration specific configurations or examples. Referring nowto the drawings, in which like numerals represent like elementsthroughout the several figures, aspects of a computing system,computer-readable storage medium, and computer-implemented methodologiesfor providing enhanced control of one or more robots, a mixedenvironment display of robotic actions, and a mixed environment displayof attached data. As will be described in more detail below with respectto FIGS. 8-10, there are a number of applications and services that canembody the functionality and techniques described herein.

FIG. 1 is a system diagram showing aspects of one illustrative mechanismdisclosed herein for providing enhanced control of one or more robots, amixed environment display of robotic actions, and a mixed environmentdisplay of attached data. As shown in FIG. 1, a system 100 may include aremote computer 101, a computing device 110, a robotic device 102, and anetwork 120. For illustrative purposes, the robotic device 102 is alsoreferred to herein as a “robot 102” or a “second computing device 102.”

The computing device 110 may operate as a stand-alone device, or thecomputing device 110 may operate in conjunction with other computers,such as the remote computer 101. As can be appreciated, the remotecomputer 101, the robot 102 and the computing device 110 areinterconnected through one or more local and/or wide area networks, suchas the network 120. In addition, the robot 102 may be in communicationwith the computing device 110 and other computers by the use of one ormore components. For instance, the robot 102 may be equipped with one ormore light sources, and the computing device 110 may include one or moresensors, including a camera, for detecting the location of the robot102. As will be described in more detail below, the robot 102 may beconfigured with light sources, sensors and transmitting devices tofacilitate communication with one or more devices. Other wired orwireless communication mechanisms may be utilized to providecommunication between one or more components and/or devices shown inFIG. 1 and other components or computers. In some configurations, therobot 102 can also include an input device, a sensor, such as a camera,or other devices for generating image data or input data 113. Any dataobtained or generated by the robot 102 can be communicated to anothercomputer or device, such as the computing device 110 or remote computer101. It should be appreciated that many more network connections may beutilized than illustrated in FIG. 1.

The computing device 110 may be in the form of a personal computer, awearable computer, including an HMD, or any other computing devicehaving components for causing a display of one or more images on adisplay, such as an interface 118. In some configurations, the interface118 may be configured to cover at least one eye of a user. In oneillustrative example, the interface 118 may include a screen configuredto cover both eyes of a user. The system 100 or aspects of the system100 may generate one or more images for generating a stereoscopic viewof one or more objects. The computing device 110 may comprise a sensor153, such as a sonar sensor, a depth sensor, infrared sensor, heatsensor, touch sensor, or any other device or component for detecting thepresence, position, and/or characteristics of an object. In addition,the computing device 110 can comprise an input device 119, such as akeyboard, mouse, microphone, or any other device configured to generatea signal and/or data based on any interaction with the computing device110. For illustrative purposes, signals or data provided by a component,such as the sensor 153 or the input device 119 is referred to herein asinput data 113. Input data 113 may also include contextual data or otherdata received from a computing system, such as the remote computer 101,or a server providing a resource or service, such as the services andresources (914-924) shown in FIG. 9.

The interface 118 may be configured to display multiple images fromdifferent environments. For example, some configurations of theinterface 118 can allow a user to see through selectable sections of theinterface 118 enabling the user to view his or her surroundings. Forillustrative purposes, the user's perspective looking through theinterface 118 is referred to herein as a “real-world view” or a “view”of an object or surrounding environment. As will be described in moredetail below, content can be displayed around selected portions of theinterface 118 enabling a user to see displayed content along with viewsof real-world objects observed through the selected portions of theinterface 118.

The configurations described herein provide both a “see through display”and an “augmented reality display.” For illustrative purposes, the “seethrough display” may include a transparent lens that can have contentdisplayed on it, and the augmented reality display may include an opaquedisplay that is configured to overlay content over a display of animage, which may be from any source, such as a video feed from a cameraused to display a real-world view. For illustrative purposes, someexamples disclosed herein describe an overlay of content over a displayof an image. In addition, some examples disclosed herein describetechniques that overlay content over a “see through display” enabling auser to see a real-world view with the content. It can be appreciatedthat all techniques described herein may apply to a “see throughdisplay,” an “augmented reality display,” or variations thereof.

The computing device 110 may include a local memory 180 that storesprofile data 103, input data 113, and application data 115. The profiledata 103 may store information describing user activity, preferences andother information used for providing control of one or more computingdevices, such as a robot. The application data 115 may include outputdata generated by techniques disclosed herein. As will be described inmore detail below, the application data 115 may include status data,image data, data identifying the geometric parameters, and other datautilized by the techniques disclosed herein.

The computing device 110 may also include a program module 111configured to manage techniques described herein and interactionsbetween a user and the computing device 110. For example, as will bedescribed in more detail below, the program module 111 may be configuredwith one or more surface reconstruction algorithms and other algorithmsfor locating objects and devices. The surface reconstruction algorithmsand other algorithms may use data or signals collected from one or moresensors 153, such as a depth sensor attached to the computing device110. The program module 111 may be in the form of a game application, avirtual reality application, an operating system component or any otherapplication configured to display image data, such as video data and/orimage data, on a display. In some illustrative examples, the programmodule 111 is a robotic controller application or game application thatcreates or allows a user to interact with a virtual world environment oran augmented reality environment. In another illustrative example, theprogram module 111 may be in the form of an operating system componentor a productivity application.

The remote computer 101 may be in the form of a server computer or anumber of server computers configured to store and process the profiledata 103, application data 115 and other information associated with theuser or other applications. As can be appreciated, the remote computer101 may store duplicate copies of the profile data 103 and theapplication data 115 allowing a centralized service to coordinate anumber of client computers, such as the computing device 110. The remotecomputer 101 may also include components, such as the server module 107,for executing one or more techniques described herein. As will bedescribed in more detail herein, the server module 107 may operate inconjunction with other modules, such as the program module 111, toimplement aspects of the techniques disclosed herein. In someconfigurations, techniques disclosed herein may be performed by thecomputing device 110.

The robot 102 may be equipped with a control module 150 for executinginstructions communicated to the robot 102. The robot 102 may have oneor more control components, such as an actuator 152. Components of therobot 102, such as the actuator 152, may be configured to generate aphysical movement of one or more objects from instructions received bythe robot 102. As will be described in more detail below, the robot 102may also comprise a number of motors configured to control the movementof the robot 102.

In some aspects of the disclosure, the computing device 110 detects oneor more conditions based on the input data 113 and other data andgenerates one or more instructions for controlling the robot 102. Insome configurations, the computing device 110 obtains input data 113 andother data describing the location and status of the robot 102. Inaddition, the computing device 110 may obtain and process dataindicating a location of the robot 102 relative to the computing device110.

Any input data 113 received from any resource, such as a remote computeror a sensor, may be used by the computing device 110 to determine thelocation of any object, the location of the computing device 110 and thelocation of the robot 102. For instance, the computing device 110 or therobot 102 may include one or more sensors for obtaining depth map data,such as a depth sensor, and other data to identify the location ofvarious objects in a room, including the room boundaries. Configurationsdisclosed herein can generate data describing geometric parameters ofany object or boundary.

Any known technology for identifying the location of one or more objectsmay be used by the techniques disclosed herein. In one example, datadefining the location of the robot 102 may be obtained by the computingdevice 110 by the use of an optical sensor, such as a camera or anyother sensor 153 or input device 119, and lights or other visualelements mounted on the robot 102. In this illustrative example,multiple LEDs may be mounted on the top of the robot 102. Some LEDs mayhave different colors to identify a direction of the robot 102. The LEDscan be mounted on the robot 102 at a predetermined distance andpredetermined position relative to one another. Knowing the distance andposition of the LEDs, in addition to other information such as the colorarrangement of the LEDs, a direction and position of the robot 102 maybe determined Any known technology, such as a technology utilizingtriangulation techniques, may be used to identify a position anddirection of the robot 102. In addition, other technologies can be usedto determine a distance between the robot 102 and the computing device110.

These examples are provided for illustrative purposes only and are notto be construed as limiting. Any technology may be used for identifyinga location of any computing device or object, which may involve the useof a radio signal, a light-based signal or any signal capable ofidentifying the location of an object. The computing device 110 mayprocess any input data 113 from any device or resource to identify thelocation and other contextual information regarding objects or computingdevices.

In some configurations, the robot 102 may have one or more sensors forcapturing and generating data. In one illustrative example, the robot102 may be equipped with one or more depth map cameras. The depth mapcameras, or any other type of sensor, may collect data describingobjects detected by the sensors. In yet another example, the robot 102may be equipped with a wheel position sensor. Data or a signal generatedby such sensors, such as the wheel position sensor, may be used toidentify the location, velocity or other information regarding the robot102. These examples are provided for illustrative purposes only and arenot to be construed as limiting. It can be appreciated that a number ofsensors or devices may be used to generate/obtain data associated withone or more objects and to identify the location of one or more objects.

The obtained data, such as depth map data, may be then processed by thetechniques described herein to identify objects and the location ofobjects, and to generate and display data associated with the object. Inthe examples described herein, the data associated with the object isdisplayed on a user interface with a representation or graphical elementthat shows an association between the data associated with the objectand an object. For illustrative purposes, data that is associated withan object is referred to herein as “attached data” or data that is“attached” to an object. In addition, any obtained data, also referredto herein as input data 113, may be used for generating and modifyinginstructions for one or more computing devices, e.g., a robot 102.

In some configurations, the computing device 110 can be configured toperform or manage complex navigation and pathfinding tasks for the robot102. Such configurations may enable the robot 102 to have a simplisticstructure and possibly simplistic software algorithms, which ultimatelyallows for a lower cost device. Such a configuration is also scalable inthat a single computing device, such as a head-mounted display, can beused to navigate and control many robots 102 without the need for eachrobot 102 to be equipped with expensive hardware or software components.

In some configurations, the computing device 110 interprets input data113 and/or other data to determine a context with respect to the objectsin the room. The computing device 110 may perform one or more functions,such as a depth map analysis and surface reconstruction analysis toidentify objects and properties of objects. For instance, certaingeometric shapes and other parameters, such as a size of an object, maybe used to categorize or characterize individual objects, e.g., anobject may be characterized as “furniture,” a “high-priority object,” ora “primary object.” Other data related to objects in an environment maybe obtained from databases or other resources, such as the services andresources depicted in FIG. 9.

In some configurations, the techniques disclosed herein may processinput data 113 from one or more resources to generate contextual data.The contextual data can be used by techniques described herein toidentify a location associated with each identified object. Based onlocation information, other data, and other properties associated witheach object, the techniques disclosed herein can generate instructionsfor a robot to perform one or more tasks. The generated instructions maybe based on the location of the identified objects, such as a computer,geometric data, characteristics of an object, and other contextualinformation.

To illustrate aspects of the techniques disclosed herein, consider ascenario where the robot 102 is in an environment, e.g., a room, withother objects. A user wearing a head-mounted display may be standing inthe room with the other objects and the robot 102. Sensors 153 and inputdevices 119 mounted to the head-mounted display can generate signals ordata associated with the robot 102 and the other objects. For instance,the signals or data can be processed by one or more methods, such astechnologies involving triangulation algorithms, to identify thelocation of the objects and/or the robot 102. Other input data 113 maybe received and processed with the signals or data to identify thelocation of the objects and/or the robot 102 and other parameters, suchas the size and shape of the objects and/or the robot 102. Processingcan be applied to any received data or signal to identify the locationand geometric properties of objects in the room. The obtainedinformation can be used to generate one or more instructions that may becommunicated to the robot 102 for execution. The instructions enable therobot 102 to perform one or more tasks, which may involve interactionbetween the robot 102 and one or more objects in the room.

With reference to FIG. 2A, a robot 102 may be configured to perform atask associated with an object 204. For example, the computing device110 may interpret various types of data, such as image data, todetermine the location of the robot 102 and the location of the object204. Based on the location of each object, the techniques disclosedherein can generate instructions for the robot 102 to interact with theobject 204. In one illustrative example, instructions configured for therobot 102 may cause the robot 102 to follow a path 201 (illustrated witha dashed line in FIG. 2A) the robot 102 may use to navigate to theobject 204. In addition, the instructions may cause the robot 102 topick up the object 204, or perform any other task, which may involve anyinteraction with the object 204. Instructions configured for the robot102 may cause the robot 102 to move along a path 201 and perform anynumber of tasks defined in the instructions, which may include one ormore tasks associated with the object 204.

During execution of the instructions, the robot 102 and other devicescan also provide feedback information to the computing device 110. Thefeedback information may be processed by the computing device 110 tomake adjustments or modifications to the generated instruction set. Asdescribed above, instructions may be generated, derived or obtainedbased on any type of input data. This feedback mechanism allows thecomputing device 110 to make adjustments to the instructionscommunicated to the robot 102 as the robot 102 is performing tasks.

FIG. 2B illustrates a scenario where an obstacle 205 is positionedbetween the robot 102 and the first object 204. Such a scenario mayarise if the obstacle 205 has been moved into the position shown in FIG.2B. As summarized above, techniques described herein enable thecomputing device 110 to adapt to changed circumstances, such as theintroduction of the obstacle 205.

By the use of one or more sensors of the computing device 110, andpossibly sensors attached to the robot 102, the computing device 110 mayidentify the location of the robot 102 relative to the obstacle 205 andthe object 204. Based on a number of parameters, which may include thesize and location of the obstacle 205, the computing device 110 maygenerate a set of instructions for the robot 102 to go around theobstacle 205 and perform one or more tasks associated with the firstobject 204. In some configurations, the generated instructions maydefine a modified path 201′ (illustrated with a dashed line in FIG. 2B)for the robot 102 to follow. In addition, the instructions may berule-based. For example, the generated instructions may simply identifyobjects to avoid. In addition, the instructions may provide specifictasks such as pick up an object, provide information, collectinformation, etc. For illustrative purposes, instructions generated inresponse to a changed circumstance or a condition, such as an error orconflict, are referred to as “modified instructions” or “modificationsto the instructions.” Adjustments to one or more instructions may bemade at any point during the execution of the instructions. Forinstance, with reference to FIG. 2B, input data 113 may be processed bythe computing device 110 at point “A,” point “B” or at any other point,to generate new instructions or modify stored instructions.

In other aspects of the present disclosure, the techniques disclosedherein provide a graphical representation of the instructions and otherstatus data. As summarized herein, it may be difficult for a user todetermine what a robot is planning and what the robot may or may not beconsidering about its surroundings. For instance, when a robot 102 isconfigured with instructions, the user is generally required to executethe instructions with the robot 102 to determine if the instructions arevalid. To address such shortcomings with respect to some existingsystems, instructions configured for a robot 102 are analyzed anddisplayed as a graphical representation.

In some configurations, the instructions are analyzed using a simulatormodeled from data describing characteristics of a mobile computingdevice, such as the robot 102. For example, if the robot 102 has amaximum speed of 2 miles per hour and can only lift 10 lbs., techniquesdisclosed herein may execute instructions using a simulator utilizingsuch parameters. In addition, a simulator can also utilize datadescribing objects or boundaries of an environment. One or moreprojected results from the simulation may be generated as output data,also referred to herein as “model data.” The techniques disclosed hereinmay then generate data defining an animation or rendering of the robot102 based on the model data. Output data defining a rendering mayinclude an animation or other graphical elements representing the modeldata. For illustrative purposes, FIGS. 3A-3E illustrate examples of adisplay that can be based on the model data and/or other data.

Consider the example scenario shown in FIG. 3A. The computing device110, which is in the form of an HMD, can provide instructions to a robot102. In this example, the HMD includes an input device 119, which caninclude a microphone, sonar sensor or any other device for receiving aninput caused by an object or user. In addition, the HMD includes asensor 153, which can include a camera directed, as illustrated by thedashed lines, toward the robot 102, an obstacle 205 or any other objectin an environment. As summarized above, the robot 102 may be configuredwith one or more visual identifiers 351, such as colored lights, for usein detecting the location and direction of the robot 102. Although thisexample illustrates a configuration with lights, any type of visualidentifier or physical characteristic may be used.

Based on a user's inspection of the instructions, it may be difficultfor the user to identify the exact path the robot 102 may travel. Inaddition, it may be difficult for the user to predict the type ofinteractions that may occur based on the instructions. To address suchissues, a graphical display 300, such as the one shown in FIG. 3B, maybe displayed to the user wearing the HMD configured with the techniquesdescribed herein. The graphical display 300 can include, for example, arepresentation of the robot 102 and representations of one or moreobjects, such as the obstacle 205. The instructions may be representedby a number of graphical elements or animations. In the example shown inFIG. 3B, based on an interpretation or a simulation of the instructions,a path 301 to be performed by the robot 102 may be determined anddisplayed. One example of the path 301 is shown with dashed line shownin FIG. 3B. As shown, the graphical display 300 can include multiplestages, where each stage can represent a task associated with theinstructions. Other representations using symbols, text descriptions,graphical elements, or animations can be displayed to show differentmovements, functions or tasks. In one illustrative example, a graphicaldisplay 300 may animate the robot 102 picking up an object, enteringdata, receiving data or any other function that may be performed by arobot. As shown, a graphical element 303 can involve text with a one ormore graphical elements, such as the quote box shown in FIG. 3B,illustrating an association between the text and the object 204.

By showing a graphical representation of a path and other tasks that maybe performed by the robot 102, a user may easily understand how therobot will interact with different objects and obstacles. By the use ofsuch technologies, instructions may be modified and tested by a user ora computer without requiring the robot 102 to actually perform theactions. Such configurations may be beneficial where fuel and otherenergy resources may be needed for a robot to operate.

These examples provided herein are for illustrative purposes only andare not to be construed as limiting. For example, some configurationsdisclosed herein may generate data that shows tasks and functions arobot has performed in the past. Thus, a history of instructionsactually executed by the robot 102 may be rendered in an animation orother type of display allowing a user to view a rendering of the robot102 with a rendering or image of the robot's environment. For example,the illustration of FIG. 3B can be used to illustrate a robot's pastperformance.

In some configurations, the graphical representation of the instructionsare displayed as “mixed reality” or “augmented reality” renderingsoverlaid on top of the real-world view entirely. The computing devicemay display the robot's instructions as well as generate instructionsand overlay the instructions on a view of the real-world where they willtake place in an environment. Renderings of the instructions orrendering describing the instructions are arranged so that they arereadily and naturally understandable to a user.

As shown in FIG. 3C and FIG. 3D, the computing device 110 can generatedifferent displays illustrating an execution or a simulation of a set ofinstructions. As shown in FIG. 3C, a position of the robot 102 relativeto an environment and an object, such as the obstacle 205, can bedisplayed to a user. In addition, a graphical element can show aremaining path 301 of the robot 102. The graphical element representingthe path 301 can show navigational projections and intent of the robot102. In addition, parts of the path 301 may be removed from the displayto indicate a stage or progress of the execution or simulation of theinstructions.

As shown in FIG. 3D, various status indicators attached to variousstages of execution can be displayed. For instance, a graphical element310 may provide status information associated with a function, task orinstruction. In the present example, the graphical element 310 indicatethat a task has been performed, e.g., that the object has been pickedup. In some configurations, the graphical element 310 includes textdescribing a function, task or instruction. The display of any statusinformation or any representation of a function task or instruction maybe based on a number of factors. For example, status information may bedisplayed when an error, conflict, or level of success is detected.

In some configurations, techniques disclosed herein may select sectionsof an interface 118, where the selected sections are configured to betransparent. Such sections provide a user with a real-world view of oneor more objects, such as the elements of an environment and the robot102. The environment may include objects and/or other aspects of theenvironment, such as walls or boundaries. For illustrative purposes, amixed environment display can include a display of graphicalrepresentations with real-world views of one or more real-world objects.For example, a rendering of a graphical element representing a path of arobot can be displayed in conjunction with a real-world view of theenvironment and/or objects in the environment.

FIG. 3E illustrates one example of a mixed environment display. In thisexample, an interface of a computing device is configured to provide oneor more transparent sections allowing users of the computing device 110to see a real-world view of objects in a surrounding environment.Configurations disclosed herein may create a selectable section 350within the interface to provide a real-world view of an object, such asthe obstacle 205, through the interface. Thus, renderings of variousobjects, such as a rendering of the robot 102, may be displayed with areal-world view of real-world objects, such as the obstacle 205. Thisexample is provided for illustrative purposes only and is not construedto be limiting. It can be appreciated that a number of transparentsections may be generated within an interface. In addition, it can beappreciated that a number of rendered objects may be displayed with anynumber of real-world views of real-world objects.

In other aspects, configurations disclosed herein determine if theinstructions create a conflict with one or more objects. If it isdetermined that the instructions create a conflict with at least oneobject, the techniques disclosed herein modify the instructions toremove or mitigate the conflict. For example, a conflict may involve aninstruction that causes the robot to collide with one or more objects ina room. The removal of the conflict may involve the generation of aninstruction that guides the robot to avoid the collision.

The new instructions may be interpreted by the system to generate agraphical representation that illustrates the execution of the newinstructions. In some configurations, graphical representations of thenew instructions may be displayed to the user in a manner shown in FIGS.3B-3D. In some configurations, a graphical representation illustratesseveral versions of different instructions so the user may view,compare, select and edit more than one set of instructions. The editsmay be performed by the use of voice commands, gestures or other actionsperformed by the user.

In other aspects of the present disclosure, by the use of the graphicalrepresentations, the computing device 110 may also illustrate a historyof commands or instructions that were executed by the robot 102. Thisdisplay of instructions may be in the form of a text description, agraphical representation or even an animation or video rendering of therobot 102. By generating and displaying a graphical representationillustrating a history of commands, a user may observe how a robot gotto a particular state. As summarized above, such graphicalrepresentations may be used for debugging and analysis purposes.

As summarized above, data describing a status or other information maybe attached to one or more components of a robot or other objects. Basedon one or more factors, the data may be displayed with a component orobject. A graphical element or the arrangement of the data with respectto a component or object may be displayed allowing the user to readilyidentify how the data is associated with one or more components orobjects.

FIG. 4A illustrates one example where data is displayed with one or moreattached objects, such as a component of the robot 102. In this example,a battery status 352 is displayed. As shown, dashed lines or othergraphical elements may be displayed to show an association between thestatus data and the attached object. In this example, the graphicalrepresentations illustrate that the battery status 352 is directlyrelated to a battery 410 and that an engine status 351 is directlyrelated to an engine 412.

FIG. 4B illustrates an example rendering of a display of attached dataindicating movement of a computing device, e.g., the robot 102. In thisexample, the velocity of a robot 102 is displayed by the use of anattached visual indicator 490. In some configurations, dashed lines orother graphical elements may be displayed to show an association betweenthe status data and the attached object. In this example, the visualindicator 490 is configured to show an association between the statusdata and the robot 102. As shown in this example, a direction andvelocity are described in the visual indicator 490. In someconfigurations, the size, position, and other characteristics of thevisual indicator 490 may be based on one or more factors. For example,the size, position, and other characteristics of the visual indicator490 may be based on the size of a view of the robot 102. This example isprovided for illustrative purposes and is not to be construed aslimiting.

FIG. 4C illustrates an example rendering of attached data with areal-world view of a computing device. In this example, an interface ofa computing device is configured to provide one or more transparentsections allowing users of the computing device to see a real-world viewof real-world objects and a surrounding environment. As shown in FIG.4C, configurations disclosed herein may create a selectable section 450within the interface to provide a real-world view of an object, such asthe robot 102, through the interface. Thus, renderings of various typesof data, such as the attached visual indicator 490, may be displayedwith a view of real-world objects, such as the robot 102. This exampleis provided for illustrative purposes only and is not construed to belimiting. It can be appreciated that any number of transparent sectionsmay be generated within an interface. In addition, it can be appreciatedthat any number of rendered graphical elements may be displayed with anynumber of real-world views of various objects.

Some configurations disclosed herein can display a graphicalaugmentation of an object on an interface. The graphical augmentationcan be positioned in relation to the position of the real-world view ofa real-world object, such as the robot 102 or any other object. Theposition of the graphical augmentation can move with the movement of thereal-world view of the object to provide the appearance that thegraphical augmentation is affixed or substantially affixed to thereal-world object. A graphical element showing a display of attacheddata can also be configured and arranged to show an association betweenthe attached data and the graphical augmentation.

FIG. 4D illustrates an example rendering of data attached to areal-world view of a computing device with the display of a mixedenvironment augmentation of the computing device. In this example,similar to the example of FIG. 4C, an interface of a computing device isconfigured to provide one or more transparent sections allowing users ofthe computing device to see a real-world view of one or more objectsand/or a surrounding environment. As shown in FIG. 4D, configurationsdisclosed herein may create a selectable section 450 within theinterface to provide a real-world view of a real-world object, such asthe robot 102, through the interface. In addition, FIG. 4D also includesthe display of a graphical representation 491 of an augmentationconfigured to supplement the real-world view of the robot 102. Theattached data, which in this example comprises a visual indicator 490,can be graphically associated with the real-world view of the robot 102or the attached data can be graphically associated with othercomponents, such as the graphical representation 491 of theaugmentation.

By showing an association between data and an attached object, a usercan readily discern a status or other information associated with one ormore devices, components, or other objects. Such configurations may bebeneficial when a robot has a large number of components. For instance,if a robot has a number of batteries or a number of motors, when anerror or problem is detected with respect to a particular battery ormotor, techniques described herein may identify the specific battery ormotor that has the problem and directly associate attached data allowingthe user to readily pinpoint any issues. As can be appreciated, anycomponent, such as the battery or engine, can be represented by anygraphical element or animation. It can be also appreciated that anygraphical element, animation, or even an arrangement or position of datamay be used to illustrate an association between data and any attachedobject.

The display of attached data may be triggered by one or more factors.For example, attached data may be displayed to a user if there is aproblem with a particular component. One or more thresholds may be usedto determine if data should be displayed to a user. In some examples, atemperature reading or a power level may be compared to a threshold andrelevant data can be displayed if one or more conditions are met. Inanother example, a usage level of a particular component can bemeasured. In such an example, a computer may store data indicating theexpected lifespan of one or more components. As the component is used,one or more measurements, e.g., a usage time measured in hours or days,may be monitored and compared to a threshold. Once one or moremeasurements approach or reach a threshold, the techniques disclosedherein may take one or more actions to indicate the same. These examplesare provided for illustrative purposes and are not to be construed aslimiting. Any type of status or measurement of any combination ofcomponents may be used to determine if attached data is to be displayed.

A status of a computing device or a status of a component of thecomputing device may be determined using one or more mechanisms. Forexample, a robot 102 may communicate the status of a component, such asthe temperature of a motor or the power level of a battery, to a remotecomputer, such as the computing device 110. In some configurations, thestatus of a computing device or the status of a component of a computingdevice may be determined by the use of a sensor. For example, a cameraof an HMD may detect a status by capturing image data of a component.Other sensors, such as an infrared sensor, may measure a temperature ofa device. Any measurement based on an analysis of image data can be usedto determine a status. For example, images may show stress fractures orparts that are out of place. In such scenarios, image data showing thedepth of a groove, a distance measured between components, or otherconditions can be used to determine a status, such as the presence of anerror or conflict.

In some configurations, contextual data describing one or more scenariosmay be used to determine when attached data is to be displayed to auser. In addition, contextual data describing the scenarios may be usedto determine the type of data that may be displayed. For example, arobot may be configured to install seats in a vehicle. When presentedwith such a scenario, the computing device may display data that isattached to the seats or certain components of the vehicle near theseats. In yet another scenario, if the robot is configured to paint thevehicle, the computing device may display data that is relevant to theexterior service of the vehicle, or the computing device may displaydata attached to the paint. By providing contextually aware, relevantinformation, user interaction with the robot and its surroundings may beimproved. The displayed data may appear and disappear automaticallydepending on any factor, such as a status of a device, a scenarioassociated with the robot, etc. In addition, the display of attacheddata may be based on a mode or a user setting of the computing device.

In an example display 400 shown in FIG. 4A, data that is attached to anobject is displayed in a manner that shows an association between thedata and the object. In addition, if the computing device detects thepresence of a person, the person may be identified by the use of facialrecognition technologies or other technologies, and based on theidentification, other information may be retrieved from one or moreresources (such as the resources depicted in FIG. 9) and displayed on aninterface. As shown in this example, such information may be attached toa rendering of the person 414 by the use of a graphical element 413. Inthis example, the person's name is displayed with a dialog box showingthe attachment between the information and a representation of theperson. Any type of information may be attached to an object, areal-world view of the person, or a rendering of the person 414. Forinstance, the attached information may describe a status of a device,person or object, or the attached information may describe othercontextual information related to the device or person.

In another example shown in FIG. 4A, a graphical element 414 can showstatus data associated with an object 204. In such an example, agraphical element 415 of the status data can describe an interactionwith the object 204. The interaction can be a past action, a current, afuture action. The graphical element 414 may also provide a descriptionof other information related to the object 204, information that may beobtained from one or more resources, such as the resources depicted inFIG. 9.

In yet another aspect of the techniques disclosed herein, a user maytrain or configure a robot. In some configurations, techniques hereinmay generate instructions or other types of commands, based on gestures,actions or other forms of input. The computing device 110 may beconfigured with a number of sensors for detecting movements and otheractions of a user. The movements and actions of the user, such asentering data on a keyboard, picking up an object, walking in a path,looking in a particular direction, or other types of activity may bedetected and translated into instructions for a robot. In someconfigurations, the instructions are conformed to guide a robot toperform the actions carried out by the user. Movements, tasks and evenan expression or a mood may be defined in the instructions. Instructionsmay allow a robot to perform, express or emulate the actions,expressions or mood captured from the user.

In some configurations, video data and other data captured by sensors orother types of input may be processed to derive contextual information.The contextual information may capture and prioritize movements andactions of a user. The contextual information may be used to determinegoals or tasks, such as to pick up an object or to move to a workstationand interact with a computer. The contextual information may also beused to generate a number of instructions which may cause a robot tofollow a particular path, direct cameras and move in a particulardirection, e.g., following the gaze direction of a user, and otheractions.

In one illustrative example, a user wearing a HMD may walk through apark and by the use of hand gestures, the user can identify trash on theground. By use of the technologies described herein, the user can pointto a particular object on the ground, and with an interpretation of thegesture and/or other types of input, such as a voice command, thecomputing device may select one or more objects and generateinstructions that may cause a robot to pick up the selected objects.

Any type of input data may be used for these techniques, which mayinvolve the use of location information from a GPS device, a map of anenvironment received from a remote computer and other contextual datasuch as the size and shape of an object. The generated instructions maybe communicated to a robot, allowing the robot to perform the tasksinterpreted by actions of the user wearing the HMD. In the presentexample, the robot may be configured to follow the user, as the userselects objects, and pick up the selected objects. In someconfigurations, the instructions may cause the robot to functionautonomously without direction from a user.

These examples are provided for illustrative purposes and not to beconstrued as limiting as any user action detected by a computing devicemay be interpreted to generate instructions for a computer or robot.FIG. 2B may be used to illustrate one example. A user wearing acomputing device such as an HMD, may follow a particular path, e.g.,from point “A” to point “B.” The user may perform different functions ateach point, such as reading information from a computer screen, enteringdata in a keyboard, communicating commands by one or more of gestures orvoice commands, etc. Instructions may be generated based on the user'sactions, which may include movement patterns, a gaze direction andinteractions with other objects or computers. In some configurations,the computing device may capture the user's actions by the use of anumber of sensors, including a camera, microphone and other sensors. Theactions may be interpreted by the computing device to generateinstructions for a computer or robot to carry out tasks or actionsperformed by the user.

The computing device may also be configured with one or moretechnologies for detecting a mood of a user or other contextualinformation. For instance, the computing device may listen toinflections in a user's voice and other observations associated with auser's expressions. The interpreted information may be used to determinea user's mood. The captured expressions, such as the user's mood, may beused to generate one or more instructions for a robot to perform, whichmay emulate the mood.

For example, in a situation where a robot is to greet people at a door,a user may wear a computing device, e.g., an HMD, to capture statementsof the user, interpret a mood of the user, and also capture otherinteractions with other people or objects. The interactions and actionsperformed by the user captured by the computing device may beinterpreted and translated into instructions for the robot. Theinstructions are configured to cause the robot to emulate the user'smood and interactions. Thus, complex tasks, such as programming a robotto greet people at a door, may be achieved by the use of natural actionsof a user.

Turning now to FIG. 5, aspects of a routine 500 for providing acontext-aware display of objects in mixed environments are shown anddescribed below. It should be understood that the operations of themethods disclosed herein are not necessarily presented in any particularorder and that performance of some or all of the operations in analternative order(s) is possible and is contemplated. The operationshave been presented in the demonstrated order for ease of descriptionand illustration. Operations may be added, omitted, and/or performedsimultaneously, without departing from the scope of the appended claims.

It also should be understood that the illustrated methods can be endedat any time and need not be performed in its entirety. Some or alloperations of the methods, and/or substantially equivalent operations,can be performed by execution of computer-readable instructions includedon a computer-storage media, as defined below. The term“computer-readable instructions,” and variants thereof, as used in thedescription and claims, is used expansively herein to include routines,applications, application modules, program modules, programs,components, data structures, algorithms, and the like. Computer-readableinstructions can be implemented on various system configurations,including single-processor or multiprocessor systems, minicomputers,mainframe computers, personal computers, hand-held computing devices,microprocessor-based, programmable consumer electronics, combinationsthereof, and the like.

Thus, it should be appreciated that the logical operations describedherein are implemented (1) as a sequence of computer implemented acts orprogram modules running on a computing system and/or (2) asinterconnected machine logic circuits or circuit modules within thecomputing system. The implementation is a matter of choice dependent onthe performance and other requirements of the computing system.Accordingly, the logical operations described herein are referred tovariously as states, operations, structural devices, acts, or modules.These operations, structural devices, acts, and modules may beimplemented in software, in firmware, in special purpose digital logic,and any combination thereof.

As will be described in conjunction with FIG. 1, the operations of theroutine 500 are described herein as being implemented, at least in part,by an application, component and/or circuit, such as the program module111 and/or the server module 107. Although the following illustrationrefers to the components of FIG. 1, it can be appreciated that theoperations of the routine 500 may be also implemented in many otherways. For example, the routine 500 may be implemented, at least in part,by computer processor or processor of another computer. In addition, oneor more of the operations of the routine 500 may alternatively oradditionally be implemented, at least in part, by a chipset workingalone or in conjunction with other software modules. Any service,circuit or application suitable for providing contextual data indicatingthe position or state of any device may be used in operations describedherein.

With reference to FIG. 5, the routine 500 begins at operation 502, wherethe program module 111 or the server module 107 obtains locationinformation for one or more objects. The location information mayidentify the location of one or more computing devices, such as a robot,and other objects surrounding the one or more computing devices. Thelocation information may also identify aspects of an environmentsurrounding the one or more computing devices, such as walls of a roomand other obstacles.

In some configurations, the location information may be obtained throughan input device of a computer, such as an HMD. Input devices, such ascameras or other sensors, may be used to identify the location of one ormore objects. In one example, an HMD worn by a user may have one or moresensors to identify the location of a robot. In addition, the sensorsmay be used to determine the location of objects surrounding the robot.As summarized above, the robot may be configured with visualidentifiers, such as colored lights, which may be detected by thesensors of the HMD. In some configurations, one or more sensors attachedto the robot may be used to collect and communicate location informationfor one or more objects.

Next, at operation 504, the one or more computing devices may obtaininstructions. The instructions may be in any format using any type oflanguage suitable for controlling a computing device, such as the robot102. In addition, the instructions may define one or more tasks for arobot to perform. For example, the instructions may be configured todirect a robot to move through one or more obstacles to interact withvarious objects in an environment. The instructions may come from anyresource, such as a local computer or a remote computer. Theinstructions may be generated by the techniques disclosed herein.

Next, at operation 506, one or more computing devices may analyze theinstructions and/or initiate a simulation of the instructions togenerate model data defining the actions or projected actions of acomputing device. In a first example, based on the location informationof the robot 102 and one or more objects, a simulation may determine apath the robot 102 may follow to avoid obstacles. In addition, asimulation may determine interactions the robot may have with one ormore obstacles such as the object 204 depicted in FIG. 3B. A simulationmay also be based on other information, such as physical characteristicsof the robot and other objects that may interact with the robot. Forinstance, the size and weight of objects, may determine if the robot iscapable of achieving the one or more tasks defined in the instructions.Operation 506 may include the generation of model data defining actionsand results of a robot based on the analysis and/or an executedsimulation of the instructions. In addition, the model data can alsoinclude performance data, such as the speed the robot is able to performthe tasks defined in the instructions.

In another example of operation 506, a robot may execute theinstructions to actually perform one or more tasks defined in theinstructions. In such an example, the robot may actually perform one ormore tasks, and generate model data based on the performance of therobot. In this example, the robot 102 and/or the computing device 110can observe and record actions performed by the robot 102 to generatemodel data. The model data can be configured to show a path the robotfollowed, performance data indicating whether the robot was able toperform one or more tasks, etc. For instance, the model data mayindicate whether a robot was actually able to avoid an obstacle, pick upa device, or carry out any other task defined in the instructions.Observations by one or more sensors 153 or input devices 119 of thecomputing device 110 may be used to determine if a task has beenachieved.

Next, at operation 508, one or more computing devices can generategraphical data based on the model data generated by the simulationand/or analysis of the instructions. The graphical data may be in anyformat and may provide a graphical representation of a task performed bythe robot or a simulation of the task defined in the instructions. Insome configurations, the graphical data may include an animationillustrating multiple stages of execution. The graphical data can beconfigured to include a graphical representation of a computing device,such as the robot 102, and/or graphical representations of one or moreobjects that interact with the computing device. For instance, thegraphical data may include one or more illustrations such as theillustrations of FIG. 3B-3E.

Next, at operation 510, one or more computing devices can cause adisplay of the graphical data on an interface. For example, thegraphical data may be displayed on an interface of a computing device,such as an HMD. The display of the graphical data can providevisualizations of a robot's actions in the space where they will occurbefore they are performed by the robot for improved understanding andsafety. Configurations disclosed herein also generate displays from arobot's perspective of the space around it, e.g., the location ofsurrounding people and objects, for improved understanding and safety.In addition, configurations disclosed herein generate data configured todisplay a robot's past actions in an environment allowing users orsystem to confirm an operation and/or to debug one or more operations.

Turning now to FIG. 6, aspects of a routine 600 for providing enhancedconfiguration and control of robots. The routine 600 begins at operation602, where one or more computing devices obtain location information forone or more objects. The location information may identify the locationof one or more computing devices, such as a robot, and other objectssurrounding the one or more computing devices. The location informationmay also identify aspects of an environment surrounding the one or morecomputing devices, such as walls of a room and other obstacles.

In some configurations, the location and parameter information may beobtained. As summarized above, information related to one or moreobjects can be obtained through an input device of a computer, such asan HMD. Input devices such as cameras or other sensors may be used toidentify the location of one or more objects. In one example, an HMDworn by a user may have one or more sensors to identify the location ofa robot. In addition, the sensors may be used to determine the locationof objects in an environment of the robot. In some configurations, therobot may be configured with identifiers, such as colored lights, whichmay be detected by the sensors of the HMD. In some configurations, oneor more sensors attached to the robot may be used to collect andcommunicate location information for one or more objects.

In one illustrative example, operation 602 and other operationsdisclosed herein can utilize the first computing device having one ormore sensors configured to detect a location and geometric parameters ofa second computing device and an environment surrounding the secondcomputing device. In some configurations, for example, the firstcomputing device may be an HMD; and the second computing device may be amobile robot. The first computing device can have a memory having afirst set of computer-executable instructions stored thereupon that cancause the first computing device to receive a signal from the sensors.Based on the signal, the first computing device can generate dataidentifying the geometric parameters of the environment. In addition,the first computing device can generate location data identifying thelocation of the second computing device relative to the environmentbased on the signal.

Next, at operation 604, one or more computing devices can cause thegeneration of a set of instructions, also referred to herein as“instructions.” In some configurations, the set of instructions can beconfigured for a second computing device, such as a robot 102, toperform one or more tasks. The set of instructions can be based, atleast in part, on the location data identifying the location of thesecond computing device and the data identifying the geometricparameters of the environment. For example, a set of instructions maydirect the robot 102 to navigate through obstacles within anenvironment. In addition, the instructions may cause the robot 102 tointeract with one or more objects, which may involve computers or otherrobots.

Next, at operation 606, the set of instructions configured for thesecond computing device can be communicated to the second computingdevice. With reference to FIG. 1, for example, a set of instructionsgenerated by the computing device 110 may be communicated to anothercomputer, such as the robot 102. This example is provided forillustrative purposes and is not to be construed as limiting. It can beappreciated that any one of the computing devices of the system 100 maygenerate instructions in any format and communicate the generatedinstructions to any computing device suitable for executing theinstructions.

Next, at operation 608, one or more computing devices can cause theexecution of the set of instructions. For example, the set ofinstructions may be executed by the robot 102. The execution of theinstructions may cause a computing device, such as the robot 102, tocomplete one or more tasks. For example, the execution of theinstructions may cause the computing device to navigate throughobstacles, interact with objects, and perform any task, e.g., action orfunction, defined in the instructions.

Next, at operation 610, one or more computing devices can detect aconflict or a changed circumstance. For illustrative purposes, aconflict may include an unintended circumstance created by the executionof the set of instructions. For instance, the computing device executingthe instructions may run into a boundary of the environment. In anotherexample, if the robot 102 is entering data at another computer, an errorin the data entry may be detected. Techniques disclosed herein may alsodetect a changed circumstance. For instance, if an object moves withinthe environment during the execution of the instructions, techniquesdisclosed herein may detect such a movement and determine that a changedcircumstance exists.

Next, at operation 612, a device, one or more computing devices cangenerate a modification to the instructions or generate new instructionsto address the conflict or the changed circumstance detected atoperation 610. With reference to the above example, where a robot 102collides with a boundary of the environment, a modification to theinstructions or new instructions can be generated to mitigate thedetected collision. With respect to the example where an object (205 ofFIG. 2B) moves with an environment, a computing device 101, by the useof its sensors, may identify the new location of the moved object.Modifications to the instructions or new instructions can be generatedto address the changed circumstance. In such an example, e.g., theinstructions generated in operation 612 may cause the robot 102 toidentify the new location of the moved object and interact with the newlocation of the object appropriately. These examples are provided forillustrative purposes and are not to be construed as limiting. Theinstructions generated in operation 612 may address any changedcircumstance, conflict, or situation.

Next, at operation 612, the modification to the instructions or newinstructions may be communicated to a computing device for execution.For example, similar to operation 606, instructions or modifications toa set of instructions may be communicated to a computing device, such asthe robot 102. Operation 612 can include the communication of newinstructions and/or data defining modifications to an instruction set.

Next at operation 614, a computing device can execute the receivedinstructions or instruction modifications. Any number of computingdevices may cause the execution of the received instructions. Withreference to the above examples, instructions received in operation 614may cause a computing device, such as the robot 102, to take correctiveactions to mitigate a conflict or address a changed circumstance.

It can be appreciated that certain operations of routine 600 may berepeated to allow a first computing device, such as an HMD, and a secondcomputing device, such as a robot, to act in concert such that the firstcomputing device may continually adjust actions taken by the secondcomputing device to address change circumstances and conflicts that mayarise during the execution of any instruction set. Any code generatedthroughout routine 600 may help a mobile computing device avoidcollisions, stay within the boundaries of an environment, and accuratelyinteract with various objects by the use of sensors of a computingdevice that is in communication with the mobile computing device.

Turning now to FIG. 7, aspects of a routine 700 for providing a mixedenvironment display of attached data. The routine 700 begins atoperation 702, where one or more computing devices obtains status data.As summarized above, the status data that may include any informationrelated to any computing device or component. For example, the statusdata that may relate to a component of the robot 102. The statusinformation may include a measurement of one or more items, such as atemperature, velocity or a direction of movement of a particularcomponent or computing device.

The status data may originate from one or more resources. For example, acamera attached to the computing device 110 may capture image data. Theimage data may be analyzed to detect a movement of one or morecomponents. Using one or more known techniques, a frame-by-frameanalysis can be used to determine the speed and direction of a movingobject. In another example, a sensor, such as an infrared sensor may beused. In such configurations, a temperature of a device or component canbe measured. In yet another example, status data may be communicatedfrom the robot 102, to one or more computing devices, such as thecomputing device 110, for further processing and analysis. In operation702, other types of data may be obtained. For example, the image datamay be analyzed to determine geometric parameters of one or moreobjects, such as the robot 102.

Next, at operation 704, an analysis of obtained data can be performed todetermine if the status data is to be displayed. One or more factors canbe used to determine if the status data is to be displayed. For example,if the temperature of a device or component reaches a threshold, thesystem can determine that such data can be displayed. Any other factor,including a user input, one or more settings or a threshold may be usedin operation 704 to determine if the status data is to be displayed.

Next, at operation 706, configurations disclosed herein cause a displayof the obtained status data. In some configurations, a graphical elementdescribing the status data may be displayed. In an illustrative example,graphical elements such as an arrow, lines, text, or other graphicalelements may be used to describe the status data or other informationobtained in operation 702. In some configurations, the displayed data isattached to one or more objects viewed on or through an interface. Forexample, an interface can be configured to provide a real-world view ofat least a portion of a computing device, such as the robot 102, througha section of the interface. A graphical element describing a status orother obtained data can be configured to indicate an association betweenthe graphical element and at least the portion of the real-world view ofthe computing device. Graphical associations between data and one ormore displayed objects may help users readily understand the context ofthe attached data.

Next, at operation 708, one or more computing devices may receive aninput. For example, an input from a user, which may include a voicecommand, a gesture detected by a camera, keyboard, a mouse, or any othertype of input, may be used to modify data or a setting of the robot 102.Operation 708, if an input is received, the routine 700 proceeds tooperation 710 where one or more computing devices modifies data based onthe input. Various settings of the robot 102 or another computing devicemay be modified based on the input. For example, if at operation 706,displayed status data indicated that the speed of a moving robot 102 is2 inches per minute, a user input may be received at operation 708controlling the speed of the robot 102. Other settings and controls canbe modified based on an input received at operation 708. After operation710 or if no input is received at operation 708, the routine 700 mayterminate or return to operation 702 where other status data may bereceived.

FIG. 8 shows additional details of an example computer architecture 800for a computer, such as the computing device 101 (FIG. 1), capable ofexecuting the program components described above for providing a mixedenvironment display of attached data. Thus, the computer architecture800 illustrated in FIG. 8 illustrates an architecture for a servercomputer, mobile phone, a PDA, a smart phone, a desktop computer, anetbook computer, a tablet computer, and/or a laptop computer. Thecomputer architecture 800 may be utilized to execute any aspects of thesoftware components presented herein.

The computer architecture 800 illustrated in FIG. 8 includes a centralprocessing unit 802 (“CPU”), a system memory 804, including a randomaccess memory 806 (“RAM”) and a read-only memory (“ROM”) 808, and asystem bus 810 that couples the memory 804 to the CPU 802. A basicinput/output system containing the basic routines that help to transferinformation between elements within the computer architecture 800, suchas during startup, is stored in the ROM 808. The computer architecture800 further includes a mass storage device 812 for storing an operatingsystem 807, and one or more application programs including, but notlimited to, a tracking module 105 and contextual data 801.

The mass storage device 812 is connected to the CPU 802 through a massstorage controller (not shown) connected to the bus 810. The massstorage device 812 and its associated computer-readable media providenon-volatile storage for the computer architecture 800. Although thedescription of computer-readable media contained herein refers to a massstorage device, such as a solid state drive, a hard disk or CD-ROMdrive, it should be appreciated by those skilled in the art thatcomputer-readable media can be any available computer storage media orcommunication media that can be accessed by the computer architecture800.

Communication media includes computer readable instructions, datastructures, program modules, or other data in a modulated data signalsuch as a carrier wave or other transport mechanism and includes anydelivery media. The term “modulated data signal” means a signal that hasone or more of its characteristics changed or set in a manner as toencode information in the signal. By way of example, and not limitation,communication media includes wired media such as a wired network ordirect-wired connection, and wireless media such as acoustic, RF,infrared and other wireless media. Combinations of the any of the aboveshould also be included within the scope of computer-readable media.

By way of example, and not limitation, computer storage media mayinclude volatile and non-volatile, removable and non-removable mediaimplemented in any method or technology for storage of information suchas computer-readable instructions, data structures, program modules orother data. For example, computer media includes, but is not limited to,RAM, ROM, EPROM, EEPROM, flash memory or other solid state memorytechnology, CD-ROM, digital versatile disks (“DVD”), HD-DVD, BLU-RAY, orother optical storage, magnetic cassettes, magnetic tape, magnetic diskstorage or other magnetic storage devices, or any other medium which canbe used to store the desired information and which can be accessed bythe computer architecture 800. For purposes the claims, the phrase“computer storage medium,” “computer-readable storage medium” andvariations thereof, does not include waves, signals, and/or othertransitory and/or intangible communication media, per se.

According to various configurations, the computer architecture 800 mayoperate in a networked environment using logical connections to remotecomputers through the network 856 and/or another network (not shown).The computer architecture 800 may connect to the network 856 through anetwork interface unit 814 connected to the bus 810. It should beappreciated that the network interface unit 814 also may be utilized toconnect to other types of networks and remote computer systems. Thecomputer architecture 800 also may include an input/output controller816 for receiving and processing input from a number of other devices,including a keyboard, mouse, or electronic stylus (not shown in FIG. 8).Similarly, the input/output controller 816 may provide output to adisplay screen, a printer, or other type of output device (also notshown in FIG. 8).

It should be appreciated that the software components described hereinmay, when loaded into the CPU 802 and executed, transform the CPU 802and the overall computer architecture 800 from a general-purposecomputing system into a special-purpose computing system customized tofacilitate the functionality presented herein. The CPU 802 may beconstructed from any number of transistors or other discrete circuitelements, which may individually or collectively assume any number ofstates. More specifically, the CPU 802 may operate as a finite-statemachine, in response to executable instructions contained within thesoftware modules disclosed herein. These computer-executableinstructions may transform the CPU 802 by specifying how the CPU 802transitions between states, thereby transforming the transistors orother discrete hardware elements constituting the CPU 802.

Encoding the software modules presented herein also may transform thephysical structure of the computer-readable media presented herein. Thespecific transformation of physical structure may depend on variousfactors, in different implementations of this description. Examples ofsuch factors may include, but are not limited to, the technology used toimplement the computer-readable media, whether the computer-readablemedia is characterized as primary or secondary storage, and the like.For example, if the computer-readable media is implemented assemiconductor-based memory, the software disclosed herein may be encodedon the computer-readable media by transforming the physical state of thesemiconductor memory. For example, the software may transform the stateof transistors, capacitors, or other discrete circuit elementsconstituting the semiconductor memory. The software also may transformthe physical state of such components in order to store data thereupon.

As another example, the computer-readable media disclosed herein may beimplemented using magnetic or optical technology. In suchimplementations, the software presented herein may transform thephysical state of magnetic or optical media, when the software isencoded therein. These transformations may include altering the magneticcharacteristics of particular locations within given magnetic media.These transformations also may include altering the physical features orcharacteristics of particular locations within given optical media, tochange the optical characteristics of those locations. Othertransformations of physical media are possible without departing fromthe scope and spirit of the present description, with the foregoingexamples provided only to facilitate this discussion.

In light of the above, it should be appreciated that many types ofphysical transformations take place in the computer architecture 800 inorder to store and execute the software components presented herein. Italso should be appreciated that the computer architecture 800 mayinclude other types of computing devices, including hand-held computers,embedded computer systems, personal digital assistants, and other typesof computing devices known to those skilled in the art. It is alsocontemplated that the computer architecture 800 may not include all ofthe components shown in FIG. 8, may include other components that arenot explicitly shown in FIG. 8, or may utilize an architecturecompletely different than that shown in FIG. 8.

FIG. 9 depicts an illustrative distributed computing environment 900capable of executing the software components described herein forproviding a mixed environment display of attached data, among otheraspects. Thus, the distributed computing environment 900 illustrated inFIG. 9 can be utilized to execute any aspects of the software componentspresented herein. For example, the distributed computing environment 900can be utilized to execute aspects of the techniques disclosed herein.

According to various implementations, the distributed computingenvironment 900 includes a computing environment 902 operating on, incommunication with, or as part of the network 904. The network 904 maybe or may include the network 856, described above with reference toFIG. 5. The network 904 also can include various access networks. One ormore client devices 906A-906N (hereinafter referred to collectivelyand/or generically as “clients 906”) can communicate with the computingenvironment 902 via the network 904 and/or other connections (notillustrated in FIG. 9). In one illustrated configuration, the clients906 include a computing device 906A such as a laptop computer, a desktopcomputer, or other computing device; a slate or tablet computing device(“tablet computing device”) 906B; a mobile computing device 906C such asa mobile telephone, a smart phone, or other mobile computing device; aserver computer 906D; and/or other devices 906N. It should be understoodthat any number of clients 906 can communicate with the computingenvironment 902. Two example computing architectures for the clients 906are illustrated and described herein with reference to FIGS. 8 and 10.It should be understood that the illustrated clients 906 and computingarchitectures illustrated and described herein are illustrative, andshould not be construed as being limited in any way.

In the illustrated configuration, the computing environment 902 includesapplication servers 908, data storage 910, and one or more networkinterfaces 912. According to various implementations, the functionalityof the application servers 908 can be provided by one or more servercomputers that are executing as part of, or in communication with, thenetwork 904. The application servers 908 can host various services,virtual machines, portals, and/or other resources. In the illustratedconfiguration, the application servers 908 host one or more virtualmachines 914 for hosting applications or other functionality. Accordingto various implementations, the virtual machines 914 host one or moreapplications and/or software modules for providing enhanced control ofone or more robots, a mixed environment display of robotic actions and amixed environment display of attached data. It should be understood thatthis configuration is illustrative, and should not be construed as beinglimiting in any way. The application servers 908 also host or provideaccess to one or more portals, link pages, Web sites, and/or otherinformation (“Web portals”) 916.

According to various implementations, the application servers 908 alsoinclude one or more mailbox services 918 and one or more messagingservices 920. The mailbox services 918 can include electronic mail(“email”) services. The mailbox services 918 also can include variouspersonal information management (“PIM”) services including, but notlimited to, calendar services, contact management services,collaboration services, and/or other services. The messaging services920 can include, but are not limited to, instant messaging services,chat services, forum services, and/or other communication services.

The application servers 908 also may include one or more socialnetworking services 922. The social networking services 922 can includevarious social networking services including, but not limited to,services for sharing or posting status updates, instant messages, links,photos, videos, and/or other information; services for commenting ordisplaying interest in articles, products, blogs, or other resources;and/or other services. In some configurations, the social networkingservices 922 are provided by or include the FACEBOOK social networkingservice, the LINKEDIN professional networking service, the MYSPACEsocial networking service, the FOURSQUARE geographic networking service,the YAMMER office colleague networking service, and the like. In otherconfigurations, the social networking services 922 are provided by otherservices, sites, and/or providers that may or may not be explicitlyknown as social networking providers. For example, some web sites allowusers to interact with one another via email, chat services, and/orother means during various activities and/or contexts such as readingpublished articles, commenting on goods or services, publishing,collaboration, gaming, and the like. Examples of such services include,but are not limited to, the WINDOWS LIVE service and the XBOX LIVEservice from Microsoft Corporation in Redmond, Wash. Other services arepossible and are contemplated.

The social networking services 922 also can include commenting,blogging, and/or micro blogging services. Examples of such servicesinclude, but are not limited to, the YELP commenting service, the KUDZUreview service, the OFFICETALK enterprise micro blogging service, theTWITTER messaging service, the GOOGLE BUZZ service, and/or otherservices. It should be appreciated that the above lists of services arenot exhaustive and that numerous additional and/or alternative socialnetworking services 922 are not mentioned herein for the sake ofbrevity. As such, the above configurations are illustrative, and shouldnot be construed as being limited in any way. According to variousimplementations, the social networking services 922 may host one or moreapplications and/or software modules for providing the functionalitydescribed herein for providing enhanced control of one or more robots, amixed environment display of robotic actions and a display of attacheddata. For instance, any one of the application servers 908 maycommunicate or facilitate the functionality and features describedherein. For instance, a social networking application, mail client,messaging client, a browser running on a phone or any other client 906may communicate with a networking service 922 and facilitate thefunctionality, even in part, described above with respect to FIGS. 5-7.

As shown in FIG. 9, the application servers 908 also can host otherservices, applications, portals, and/or other resources (“otherresources”) 924. The other resources 924 can include, but are notlimited to, document sharing, rendering or any other functionality. Itthus can be appreciated that the computing environment 902 can provideintegration of the concepts and technologies disclosed herein providedherein with various mailbox, messaging, social networking, and/or otherservices or resources.

As mentioned above, the computing environment 902 can include the datastorage 910. According to various implementations, the functionality ofthe data storage 910 is provided by one or more databases operating on,or in communication with, the network 904. The functionality of the datastorage 910 also can be provided by one or more server computersconfigured to host data for the computing environment 902. The datastorage 910 can include, host, or provide one or more real or virtualdatastores 926A-926N (hereinafter referred to collectively and/orgenerically as “datastores 926”). The datastores 926 are configured tohost data used or created by the application servers 908 and/or otherdata. Although not illustrated in FIG. 9, the datastores 926 also canhost or store web page documents, word documents, presentationdocuments, data structures, algorithms for execution by a recommendationengine, and/or other data utilized by any application program or anothermodule, such as the content manager 105. Aspects of the datastores 926may be associated with a service for storing files.

The computing environment 902 can communicate with, or be accessed by,the network interfaces 912. The network interfaces 912 can includevarious types of network hardware and software for supportingcommunications between two or more computing devices including, but notlimited to, the clients 906 and the application servers 908. It shouldbe appreciated that the network interfaces 912 also may be utilized toconnect to other types of networks and/or computer systems.

It should be understood that the distributed computing environment 900described herein can provide any aspects of the software elementsdescribed herein with any number of virtual computing resources and/orother distributed computing functionality that can be configured toexecute any aspects of the software components disclosed herein.According to various implementations of the concepts and technologiesdisclosed herein, the distributed computing environment 900 provides thesoftware functionality described herein as a service to the clients 906.It should be understood that the clients 906 can include real or virtualmachines including, but not limited to, server computers, web servers,personal computers, mobile computing devices, smart phones, and/or otherdevices. As such, various configurations of the concepts andtechnologies disclosed herein enable any device configured to access thedistributed computing environment 900 to utilize the functionalitydescribed herein for providing enhanced control of one or more robots, amixed environment display of robotic actions, a display of attacheddata, among other aspects. In one specific example, as summarized above,techniques described herein may be implemented, at least in part, by theoperating system 807 of FIG. 8, which works in conjunction with theapplication servers 908 of FIG. 9.

Turning now to FIG. 10, an illustrative computing device architecture1000 for a computing device that is capable of executing varioussoftware components described herein for providing enhanced control ofone or more robots, a mixed environment display of robotic actions and adisplay of attached data. The computing device architecture 1000 isapplicable to computing devices that facilitate mobile computing due, inpart, to form factor, wireless connectivity, and/or battery-poweredoperation. In some configurations, the computing devices include, butare not limited to, mobile telephones, tablet devices, slate devices,portable video game devices, and the like. The computing devicearchitecture 1000 is applicable to any of the clients 806 shown in FIG.8. Moreover, aspects of the computing device architecture 1000 may beapplicable to traditional desktop computers, portable computers (e.g.,laptops, notebooks, ultra-portables, and netbooks), server computers,and other computer systems, such as those described herein. For example,the single touch and multi-touch aspects disclosed herein below may beapplied to desktop computers that utilize a touchscreen or some othertouch-enabled device, such as a touch-enabled track pad or touch-enabledmouse.

The computing device architecture 1000 illustrated in FIG. 10 includes aprocessor 1002, memory components 1004, network connectivity components1006, sensor components 1008, input/output components 1010, and powercomponents 1012. In the illustrated configuration, the processor 1002 isin communication with the memory components 1004, the networkconnectivity components 1006, the sensor components 1008, theinput/output (“I/O”) components 1010, and the power components 1012.Although no connections are shown between the individuals componentsillustrated in FIG. 10, the components can interact to carry out devicefunctions. In some configurations, the components are arranged so as tocommunicate via one or more busses (not shown).

The processor 1002 includes a central processing unit (“CPU”) configuredto process data, execute computer-executable instructions of one or moreapplication programs, and communicate with other components of thecomputing device architecture 1000 in order to perform variousfunctionality described herein. The processor 1002 may be utilized toexecute aspects of the software components presented herein and,particularly, those that utilize, at least in part, a touch-enabledinput.

In some configurations, the processor 1002 includes a graphicsprocessing unit (“GPU”) configured to accelerate operations performed bythe CPU, including, but not limited to, operations performed byexecuting general-purpose scientific and/or engineering computingapplications, as well as graphics-intensive computing applications suchas high resolution video (e.g., 720P, 1080P, and higher resolution),video games, three-dimensional (“3D”) modeling applications, and thelike. In some configurations, the processor 1002 is configured tocommunicate with a discrete GPU (not shown). In any case, the CPU andGPU may be configured in accordance with a co-processing CPU/GPUcomputing model, wherein the sequential part of an application executeson the CPU and the computationally-intensive part is accelerated by theGPU.

In some configurations, the processor 1002 is, or is included in, asystem-on-chip (“SoC”) along with one or more of the other componentsdescribed herein below. For example, the SoC may include the processor1002, a GPU, one or more of the network connectivity components 1006,and one or more of the sensor components 1008. In some configurations,the processor 1002 is fabricated, in part, utilizing apackage-on-package (“PoP”) integrated circuit packaging technique. Theprocessor 1002 may be a single core or multi-core processor.

The processor 1002 may be created in accordance with an ARMarchitecture, available for license from ARM HOLDINGS of Cambridge,United Kingdom. Alternatively, the processor 1002 may be created inaccordance with an x86 architecture, such as is available from INTELCORPORATION of Mountain View, Calif. and others. In some configurations,the processor 1002 is a SNAPDRAGON SoC, available from QUALCOMM of SanDiego, Calif., a TEGRA SoC, available from NVIDIA of Santa Clara,Calif., a HUMMINGBIRD SoC, available from SAMSUNG of Seoul, South Korea,an Open Multimedia Application Platform (“OMAP”) SoC, available fromTEXAS INSTRUMENTS of Dallas, Tex., a customized version of any of theabove SoCs, or a proprietary SoC.

The memory components 1004 include a random access memory (“RAM”) 1014,a read-only memory (“ROM”) 1016, an integrated storage memory(“integrated storage”) 1018, and a removable storage memory (“removablestorage”) 1020. In some configurations, the RAM 1014 or a portionthereof, the ROM 1016 or a portion thereof, and/or some combination theRAM 1014 and the ROM 1016 is integrated in the processor 1002. In someconfigurations, the ROM 1016 is configured to store a firmware, anoperating system or a portion thereof (e.g., operating system kernel),and/or a bootloader to load an operating system kernel from theintegrated storage 1018 and/or the removable storage 1020.

The integrated storage 1018 can include a solid-state memory, a harddisk, or a combination of solid-state memory and a hard disk. Theintegrated storage 1018 may be soldered or otherwise connected to alogic board upon which the processor 1002 and other components describedherein also may be connected. As such, the integrated storage 1018 isintegrated in the computing device. The integrated storage 1018 isconfigured to store an operating system or portions thereof, applicationprograms, data, and other software components described herein.

The removable storage 1020 can include a solid-state memory, a harddisk, or a combination of solid-state memory and a hard disk. In someconfigurations, the removable storage 1020 is provided in lieu of theintegrated storage 1018. In other configurations, the removable storage1020 is provided as additional optional storage. In some configurations,the removable storage 1020 is logically combined with the integratedstorage 1018 such that the total available storage is made available asa total combined storage capacity. In some configurations, the totalcombined capacity of the integrated storage 1018 and the removablestorage 1020 is shown to a user instead of separate storage capacitiesfor the integrated storage 1018 and the removable storage 1020.

The removable storage 1020 is configured to be inserted into a removablestorage memory slot (not shown) or other mechanism by which theremovable storage 1020 is inserted and secured to facilitate aconnection over which the removable storage 1020 can communicate withother components of the computing device, such as the processor 1002.The removable storage 1020 may be embodied in various memory cardformats including, but not limited to, PC card, CompactFlash card,memory stick, secure digital (“SD”), miniSD, microSD, universalintegrated circuit card (“UICC”) (e.g., a subscriber identity module(“SIM”) or universal SIM (“USIM”)), a proprietary format, or the like.

It can be understood that one or more of the memory components 1004 canstore an operating system. According to various configurations, theoperating system includes, but is not limited to WINDOWS MOBILE OS fromMicrosoft Corporation of Redmond, Wash., WINDOWS PHONE OS from MicrosoftCorporation, WINDOWS from Microsoft Corporation, PALM WEBOS fromHewlett-Packard Company of Palo Alto, Calif., BLACKBERRY OS fromResearch In Motion Limited of Waterloo, Ontario, Canada, IOS from AppleInc. of Cupertino, Calif., and ANDROID OS from Google Inc. of MountainView, Calif. Other operating systems are contemplated.

The network connectivity components 1006 include a wireless wide areanetwork component (“WWAN component”) 1022, a wireless local area networkcomponent (“WLAN component”) 1024, and a wireless personal area networkcomponent (“WPAN component”) 1026. The network connectivity components1006 facilitate communications to and from the network 856 or anothernetwork, which may be a WWAN, a WLAN, or a WPAN. Although only thenetwork 856 is illustrated, the network connectivity components 1006 mayfacilitate simultaneous communication with multiple networks, includingthe network 704 of FIG. 7. For example, the network connectivitycomponents 1006 may facilitate simultaneous communications with multiplenetworks via one or more of a WWAN, a WLAN, or a WPAN.

The network 856 may be or may include a WWAN, such as a mobiletelecommunications network utilizing one or more mobiletelecommunications technologies to provide voice and/or data services toa computing device utilizing the computing device architecture 1000 viathe WWAN component 1022. The mobile telecommunications technologies caninclude, but are not limited to, Global System for Mobile communications(“GSM”), Code Division Multiple Access (“CDMA”) ONE, CDMA7000, UniversalMobile Telecommunications System (“UMTS”), Long Term Evolution (“LTE”),and Worldwide Interoperability for Microwave Access (“WiMAX”). Moreover,the network 856 may utilize various channel access methods (which may ormay not be used by the aforementioned standards) including, but notlimited to, Time Division Multiple Access (“TDMA”), Frequency DivisionMultiple Access (“FDMA”), CDMA, wideband CDMA (“W-CDMA”), OrthogonalFrequency Division Multiplexing (“OFDM”), Space Division Multiple Access(“SDMA”), and the like. Data communications may be provided usingGeneral Packet Radio Service (“GPRS”), Enhanced Data rates for GlobalEvolution (“EDGE”), the High-Speed Packet Access (“HSPA”) protocolfamily including High-Speed Downlink Packet Access (“HSDPA”), EnhancedUplink (“EUL”) or otherwise termed High-Speed Uplink Packet Access(“HSUPA”), Evolved HSPA (“HSPA+”), LTE, and various other current andfuture wireless data access standards. The network 856 may be configuredto provide voice and/or data communications with any combination of theabove technologies. The network 856 may be configured to or adapted toprovide voice and/or data communications in accordance with futuregeneration technologies.

In some configurations, the WWAN component 1022 is configured to providedual-multi-mode connectivity to the network 856. For example, the WWANcomponent 1022 may be configured to provide connectivity to the network856, wherein the network 856 provides service via GSM and UMTStechnologies, or via some other combination of technologies.Alternatively, multiple WWAN components 1022 may be utilized to performsuch functionality, and/or provide additional functionality to supportother non-compatible technologies (i.e., incapable of being supported bya single WWAN component). The WWAN component 1022 may facilitate similarconnectivity to multiple networks (e.g., a UMTS network and an LTEnetwork).

The network 856 may be a WLAN operating in accordance with one or moreInstitute of Electrical and Electronic Engineers (“IEEE”) 802.11standards, such as IEEE 802.11a, 802.11b, 802.11g, 802.11n, and/orfuture 802.11 standard (referred to herein collectively as WI-FI). Draft802.11 standards are also contemplated. In some configurations, the WLANis implemented utilizing one or more wireless WI-FI access points. Insome configurations, one or more of the wireless WI-FI access points areanother computing device with connectivity to a WWAN that arefunctioning as a WI-FI hotspot. The WLAN component 1024 is configured toconnect to the network 856 via the WI-FI access points. Such connectionsmay be secured via various encryption technologies including, but notlimited, WI-FI Protected Access (“WPA”), WPA2, Wired Equivalent Privacy(“WEP”), and the like.

The network 856 may be a WPAN operating in accordance with Infrared DataAssociation (“IrDA”), BLUETOOTH, wireless Universal Serial Bus (“USB”),Z-Wave, ZIGBEE, or some other short-range wireless technology. In someconfigurations, the WPAN component 1026 is configured to facilitatecommunications with other devices, such as peripherals, computers, orother computing devices via the WPAN.

The sensor components 1008 include a magnetometer 1028, an ambient lightsensor 1030, a proximity sensor 1032, an accelerometer 1034, a gyroscope1036, and a Global Positioning System sensor (“GPS sensor”) 1038. It iscontemplated that other sensors, such as, but not limited to,temperature sensors or shock detection sensors, also may be incorporatedin the computing device architecture 1000.

The magnetometer 1028 is configured to measure the strength anddirection of a magnetic field. In some configurations the magnetometer1028 provides measurements to a compass application program storedwithin one of the memory components 1004 in order to provide a user withaccurate directions in a frame of reference including the cardinaldirections, north, south, east, and west. Similar measurements may beprovided to a navigation application program that includes a compasscomponent. Other uses of measurements obtained by the magnetometer 1028are contemplated.

The ambient light sensor 1030 is configured to measure ambient light. Insome configurations, the ambient light sensor 1030 provides measurementsto an application program stored within one of the memory components1004 in order to automatically adjust the brightness of a display(described below) to compensate for low-light and high-lightenvironments. Other uses of measurements obtained by the ambient lightsensor 1030 are contemplated.

The proximity sensor 1032 is configured to detect the presence of anobject or thing in proximity to the computing device without directcontact. In some configurations, the proximity sensor 1032 detects thepresence of a user's body (e.g., the user's face) and provides thisinformation to an application program stored within one of the memorycomponents 1004 that utilizes the proximity information to enable ordisable some functionality of the computing device. For example, atelephone application program may automatically disable a touchscreen(described below) in response to receiving the proximity information sothat the user's face does not inadvertently end a call or enable/disableother functionality within the telephone application program during thecall. Other uses of proximity as detected by the proximity sensor 1032are contemplated.

The accelerometer 1034 is configured to measure proper acceleration. Insome configurations, output from the accelerometer 1034 is used by anapplication program as an input mechanism to control some functionalityof the application program. For example, the application program may bea video game in which a character, a portion thereof, or an object ismoved or otherwise manipulated in response to input received via theaccelerometer 1034. In some configurations, output from theaccelerometer 1034 is provided to an application program for use inswitching between landscape and portrait modes, calculating coordinateacceleration, or detecting a fall. Other uses of the accelerometer 1034are contemplated.

The gyroscope 1036 is configured to measure and maintain orientation. Insome configurations, output from the gyroscope 1036 is used by anapplication program as an input mechanism to control some functionalityof the application program. For example, the gyroscope 1036 can be usedfor accurate recognition of movement within a 3D environment of a videogame application or some other application. In some configurations, anapplication program utilizes output from the gyroscope 1036 and theaccelerometer 1034 to enhance control of some functionality of theapplication program. Other uses of the gyroscope 1036 are contemplated.

The GPS sensor 1038 is configured to receive signals from GPS satellitesfor use in calculating a location. The location calculated by the GPSsensor 1038 may be used by any application program that requires orbenefits from location information. For example, the location calculatedby the GPS sensor 1038 may be used with a navigation application programto provide directions from the location to a destination or directionsfrom the destination to the location. Moreover, the GPS sensor 1038 maybe used to provide location information to an external location-basedservice, such as E911 service. The GPS sensor 1038 may obtain locationinformation generated via WI-FI, WIMAX, and/or cellular triangulationtechniques utilizing one or more of the network connectivity components1006 to aid the GPS sensor 1038 in obtaining a location fix. The GPSsensor 1038 may also be used in Assisted GPS (“A-GPS”) systems.

The I/O components 1010 include a display 1040, a touchscreen 1042, adata I/O interface component (“data I/O”) 1044, an audio I/O interfacecomponent (“audio I/O”) 1046, a video I/O interface component (“videoI/O”) 1048, and a camera 1050. In some configurations, the display 1040and the touchscreen 1042 are combined. In some configurations two ormore of the data I/O component 1044, the audio I/O component 1046, andthe video I/O component 1048 are combined. The I/O components 1010 mayinclude discrete processors configured to support the various interfacedescribed below, or may include processing functionality built-in to theprocessor 1002.

The display 1040 is an output device configured to present informationin a visual form. In particular, the display 1040 may present graphicaluser interface (“GUI”) elements, text, images, video, notifications,virtual buttons, virtual keyboards, messaging data, Internet content,device status, time, date, calendar data, preferences, map information,location information, and any other information that is capable of beingpresented in a visual form. In some configurations, the display 1040 isa liquid crystal display (“LCD”) utilizing any active or passive matrixtechnology and any backlighting technology (if used). In someconfigurations, the display 1040 is an organic light emitting diode(“OLED”) display. Other display types are contemplated.

The touchscreen 1042, also referred to herein as a “touch-enabledscreen,” is an input device configured to detect the presence andlocation of a touch. The touchscreen 1042 may be a resistivetouchscreen, a capacitive touchscreen, a surface acoustic wavetouchscreen, an infrared touchscreen, an optical imaging touchscreen, adispersive signal touchscreen, an acoustic pulse recognitiontouchscreen, or may utilize any other touchscreen technology. In someconfigurations, the touchscreen 1042 is incorporated on top of thedisplay 1040 as a transparent layer to enable a user to use one or moretouches to interact with objects or other information presented on thedisplay 1040. In other configurations, the touchscreen 1042 is a touchpad incorporated on a surface of the computing device that does notinclude the display 1040. For example, the computing device may have atouchscreen incorporated on top of the display 1040 and a touch pad on asurface opposite the display 1040.

In some configurations, the touchscreen 1042 is a single-touchtouchscreen. In other configurations, the touchscreen 1042 is amulti-touch touchscreen. In some configurations, the touchscreen 1042 isconfigured to detect discrete touches, single touch gestures, and/ormulti-touch gestures. These are collectively referred to herein asgestures for convenience. Several gestures will now be described. Itshould be understood that these gestures are illustrative and are notintended to limit the scope of the appended claims. Moreover, thedescribed gestures, additional gestures, and/or alternative gestures maybe implemented in software for use with the touchscreen 1042. As such, adeveloper may create gestures that are specific to a particularapplication program.

In some configurations, the touchscreen 1042 supports a tap gesture inwhich a user taps the touchscreen 1042 once on an item presented on thedisplay 1040. The tap gesture may be used for various reasons including,but not limited to, opening or launching whatever the user taps. In someconfigurations, the touchscreen 1042 supports a double tap gesture inwhich a user taps the touchscreen 1042 twice on an item presented on thedisplay 1040. The double tap gesture may be used for various reasonsincluding, but not limited to, zooming in or zooming out in stages. Insome configurations, the touchscreen 1042 supports a tap and holdgesture in which a user taps the touchscreen 1042 and maintains contactfor at least a pre-defined time. The tap and hold gesture may be usedfor various reasons including, but not limited to, opening acontext-specific menu.

In some configurations, the touchscreen 1042 supports a pan gesture inwhich a user places a finger on the touchscreen 1042 and maintainscontact with the touchscreen 1042 while moving the finger on thetouchscreen 1042. The pan gesture may be used for various reasonsincluding, but not limited to, moving through screens, images, or menusat a controlled rate. Multiple finger pan gestures are alsocontemplated. In some configurations, the touchscreen 1042 supports aflick gesture in which a user swipes a finger in the direction the userwants the screen to move. The flick gesture may be used for variousreasons including, but not limited to, scrolling horizontally orvertically through menus or pages. In some configurations, thetouchscreen 1042 supports a pinch and stretch gesture in which a usermakes a pinching motion with two fingers (e.g., thumb and forefinger) onthe touchscreen 1042 or moves the two fingers apart. The pinch andstretch gesture may be used for various reasons including, but notlimited to, zooming gradually in or out of a website, map, or picture.

Although the above gestures have been described with reference to theuse one or more fingers for performing the gestures, other appendagessuch as toes or objects such as styluses may be used to interact withthe touchscreen 1042. As such, the above gestures should be understoodas being illustrative and should not be construed as being limiting inany way.

The data I/O interface component 1044 is configured to facilitate inputof data to the computing device and output of data from the computingdevice. In some configurations, the data I/O interface component 1044includes a connector configured to provide wired connectivity betweenthe computing device and a computer system, for example, forsynchronization operation purposes. The connector may be a proprietaryconnector or a standardized connector such as USB, micro-USB, mini-USB,or the like. In some configurations, the connector is a dock connectorfor docking the computing device with another device such as a dockingstation, audio device (e.g., a digital music player), or video device.

The audio I/O interface component 1046 is configured to provide audioinput and/or output capabilities to the computing device. In someconfigurations, the audio I/O interface component 1046 includes amicrophone configured to collect audio signals. In some configurations,the audio I/O interface component 1046 includes a headphone jackconfigured to provide connectivity for headphones or other externalspeakers. In some configurations, the audio I/O interface component 1046includes a speaker for the output of audio signals. In someconfigurations, the audio I/O interface component 1046 includes anoptical audio cable out.

The video I/O interface component 1048 is configured to provide videoinput and/or output capabilities to the computing device. In someconfigurations, the video I/O interface component 1048 includes a videoconnector configured to receive video as input from another device(e.g., a video media player such as a DVD or BLURAY player) or sendvideo as output to another device (e.g., a monitor, a television, orsome other external display). In some configurations, the video I/Ointerface component 1048 includes a High-Definition Multimedia Interface(“HDMI”), mini-HDMI, micro-HDMI, DisplayPort, or proprietary connectorto input/output video content. In some configurations, the video I/Ointerface component 1048 or portions thereof is combined with the audioI/O interface component 1046 or portions thereof.

The camera 1050 can be configured to capture still images and/or video.The camera 1050 may utilize a charge coupled device (“CCD”) or acomplementary metal oxide semiconductor (“CMOS”) image sensor to captureimages. In some configurations, the camera 1050 includes a flash to aidin taking pictures in low-light environments. Settings for the camera1050 may be implemented as hardware or software buttons.

Although not illustrated, one or more hardware buttons may also beincluded in the computing device architecture 1000. The hardware buttonsmay be used for controlling some operational aspect of the computingdevice. The hardware buttons may be dedicated buttons or multi-usebuttons. The hardware buttons may be mechanical or sensor-based.

The illustrated power components 1012 include one or more batteries1052, which can be connected to a battery gauge 1054. The batteries 1052may be rechargeable or disposable. Rechargeable battery types include,but are not limited to, lithium polymer, lithium ion, nickel cadmium,and nickel metal hydride. Each of the batteries 1052 may be made of oneor more cells.

The battery gauge 1054 can be configured to measure battery parameterssuch as current, voltage, and temperature. In some configurations, thebattery gauge 1054 is configured to measure the effect of a battery'sdischarge rate, temperature, age and other factors to predict remaininglife within a certain percentage of error. In some configurations, thebattery gauge 1054 provides measurements to an application program thatis configured to utilize the measurements to present useful powermanagement data to a user. Power management data may include one or moreof a percentage of battery used, a percentage of battery remaining, abattery condition, a remaining time, a remaining capacity (e.g., in watthours), a current draw, and a voltage.

The power components 1012 may also include a power connector, which maybe combined with one or more of the aforementioned I/O components 1010.The power components 1012 may interface with an external power system orcharging equipment via an I/O component.

Based on the foregoing, it should be appreciated that concepts andtechnologies have been disclosed herein that provide, among othertechniques, enhanced control of one or more robots. Although the subjectmatter presented herein has been described in language specific tocomputer structural features, methodological and transformative acts,specific computing machinery, and computer readable media, it is to beunderstood that the invention defined in the appended claims is notnecessarily limited to the specific features, acts, or media describedherein. Rather, the specific features, acts and mediums are disclosed asexample forms of implementing the claims.

The subject matter described above is provided by way of illustrationonly and should not be construed as limiting. Various modifications andchanges may be made to the subject matter described herein withoutfollowing the example configurations and applications illustrated anddescribed, and without departing from the true spirit and scope of thepresent invention, which is set forth in the following claims.

What is claimed is:
 1. A computer-implemented method, comprising:obtaining status data, at a first computing device, indicating a statusassociated with a second computing device; determining if the status isto be displayed; and if the status is to be displayed, generating agraphical element describing the status, and causing a display of thegraphical element describing the status on an interface of the firstcomputing device, wherein the interface is configured to provide areal-world view of at least a portion of the second computing device andwherein the graphical element describing the status is configured toindicate an association between the graphical element describing thestatus and at least the portion of the real-world view of the secondcomputing device.
 2. The method of claim 1, wherein the first computingdevice is a head-mounted display comprising at least one image cameraconfigured to generate image data, and wherein obtaining the status datacomprises analyzing the image data to generate the status data.
 3. Themethod of claim 1, wherein the first computing device is a head-mounteddisplay comprising at least one image camera configured to generateimage data, and wherein the method further comprises: analyzing theimage data to determine geometric parameters of at least one componentof the second computing device; and determining a size or a position ofthe graphical element describing the status based, at least in part, onthe geometric parameters of at least one component of the secondcomputing device.
 4. The computer-implemented method of claim 1, whereinthe status indicates a characteristic of a movement of the secondcomputing device.
 5. The computer-implemented method of claim 1, whereinthe status indicates a characteristic of an interaction between thesecond computing device and at least one object.
 6. Thecomputer-implemented method of claim 1, wherein the second computingdevice is configured to perform one or more tasks defined in a set ofinstructions, and wherein the display of the graphical element includesa description of an individual task of one or more tasks defined in theset of instructions.
 7. The computer-implemented method of claim 1,wherein the computer-implemented method further comprises: causing ageneration of a graphical augmentation of the second computing device;and causing a display of the graphical augmentation on the interface,wherein the graphical augmentation is positioned in relation to aposition of the real-world view of the second computing device throughthe interface, and wherein an arrangement of the display indicates anassociation between the graphical element and the graphical augmentationof the second computing device.
 8. The computer-implemented method ofclaim 1, wherein the computer-implemented method further comprises:determining a status of a component of the second computing device; andcausing a display of a graphical element describing the status of thecomponent on the interface, wherein the graphical element describing thestatus of the component is configured to indicate an association betweenthe graphical element describing the status of the component and atleast a portion of the real-world view of the second computing device.9. The computer-implemented method of claim 1, wherein thecomputer-implemented method further comprises: determining if a statusof a component of the second computing device exceeds a threshold; andif the status of the component of the second computing device exceedsthe threshold, causing a display of a graphical element indicating thestatus of the component on the interface, wherein the graphical elementindicating the status of the component is configured to indicate anassociation between the graphical element indicating the status of thecomponent and at least a portion of the real-world view of the secondcomputing device.
 10. The computer-implemented method of claim 1,wherein the second computing device is configured to perform one or moretasks defined in a set of instructions, and wherein the instructionsdefine interactions between the second computing device and a real-worldobject, and wherein the method further comprises: receiving a signalfrom one or more sensors attached to the second computing device,wherein the one or more sensors are configured to detect a status of thereal-world object; causing a generation of a graphical elementindicating the status of the real-world object; and causing a display ofthe graphical element indicating the status of the real-world object onthe interface, wherein the graphical element indicating the status ofthe real-world object is configured to illustrate an association betweenthe graphical element indicating the status of the real-world object anda real-world view of the real-world object or a graphical representationof the real-world object.
 11. A first computing device, comprising: aprocessor; an interface; a memory having a set of computer-executableinstructions stored thereupon which, when executed by the processor,cause the first computing device to determine if a status associatedwith a component of a second computing device reaches a threshold; andif the status reaches the threshold, generate a graphical elementdescribing the status and a graphical representation of the component,and cause a display of the graphical representation of the status on theinterface and a view of the component on or through the interface,wherein an arrangement of the display indicates an association betweenthe graphical representation of the status and the view of thecomponent.
 12. The first computing device of claim 11, wherein the viewof the component is a graphical representation of the componentdisplayed on the interface.
 13. The first computing device of claim 11,wherein the interface is configured to provide a real-world view of thecomponent through a section of the interface, and wherein the view ofthe component is a real-world view of the component through the sectionof the interface.
 14. The first computing device of claim 11, whereinthe status comprises a power level.
 15. The first computing device ofclaim 11, wherein the status comprises a temperature level.
 16. Thefirst computing device of claim 11, wherein the status comprises a usagelevel.
 17. A computer storage medium having a first set ofcomputer-executable instructions stored thereupon which, when executedby a first computing device, cause the first computing device to: obtainstatus data indicating a status associated with a second computingdevice performing one or more tasks defined in a second set ofinstructions; determine if the status is to be displayed; and if thestatus is to be displayed, generate a graphical element describing thestatus, and cause a display of the graphical element describing thestatus on an interface of the first computing device, wherein theinterface is configured to provide a view of at least a portion of thesecond computing device, and wherein the graphical element is configuredto indicate an association between the graphical element describing thestatus and at least the portion of the view of the second computingdevice.
 18. The computer storage medium of claim 17, wherein the view ofthe second computing device is a graphical representation of the secondcomputing device displayed on the interface.
 19. The computer storagemedium of claim 17, wherein the interface is configured to provide areal-world view of the second computing device through a section of theinterface, and wherein view of the second computing device is areal-world view of the second computing device through the section ofthe interface.
 20. The computer storage medium of claim 17, whereinfirst set of computer-executable instructions cause the first computingdevice to: receive an input in response to the display of the graphicalelement describing the status; and modifying one or more settings of thesecond computing device in response to the input.